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Abstract 

The  hypothesis  of  this  paper  is  as  follows:  Uncertainty  and  inconsistency  during  complex 
endeavors  can  be  reduced  through  the  application  of  augmented  cognition.  An  analysis  of  the 
genealogy  of  modem  decision  aides  leads  one  to  conclude  that  we  should  only  be  discussing  the 
capability  spectrum  of  intelligent  software  agents.  We  believe  that  this  represents  a  limited  view 
of  the  field  of  automated  decision  aides  and  assisted  cognition.  Instead  of  asking  how  smart  the 
software  agents  can  become,  we  would  like  to  propose  the  following  question:  Can  we  make  the 
human  being  smarter?  Is  it  possible  to  improve  cognitive  functions  inside  the  mind  resulting  in 
better  selections  of  decision  alternatives  and  interpretations  of  events?  Is  it  possible  to  radically 
alter  human  training,  development,  and  education  to  optimize  the  potential  of  every  individual? 
The  authors  believe  that  the  processes  and  tools  being  developed  in  the  emerging  field  of 
augmented  cognition  1  can  be  exploited  to  provide  a  novel  fusion  of  more  capable  human  beings 
and  exotic  software  agents.  This  fusion  should  result  in  breakthrough  levels  of  situational 
awareness  and  superior  decision  making  in  environments  of  extreme  uncertainty. 
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Introduction 


In  this  paper  we  describe  augmented  cognition  research  that  we  believe  could  be  exploited  to 
reduce  uncertainty  and  inconsistency  in  military  planning  and  decision  making.  Augmented 
cognition  speaks  of  directly  connecting  parts  of  the  brain  to  external  interfaces  where  agents  can 
monitor  internal  cognitive  states,  create  ‘external  mental  models’,  evaluate  those  models  and 
states  and  request  a  re-prioritization  of  certain  cognitive  and  mental  activities  back  to  the  brain  of 
the  human  user.  We  consider  agents  capable  of  modifying  internal  human  cognitive  processes  to 
be  exotic  by  definition.  Uncertainty  in  decision  making  has  been  analyzed  in  some  detail.  The 
two  types  of  uncertainty  which  we  are  interested  in  reducing  are  aleatoric  and  epistemic.  While 
only  multiple,  diverse,  and  rich  courses  of  action  can  manage  aleatoric  uncertainty,  epistemic 
uncertainty  arises  from  the  quality  of  a  model;  that  is,  the  extent  to  which  a  model,  because  of  its 
necessary  assumptions  and  approximations,  does  not  truly  represent  that  which  is  being 
modeled2.  Situational  awareness,  a  key  aspect  of  decision  making,  is  based  upon  many  models, 
for  example,  a  common  operational  picture,  a  common  tactical  picture,  etc.  all  of  which  are 
based  upon  the  assumption  of  timeliness,  validity,  and  understandability  of  track  data  and  its 
sources.  But  above  all,  the  common  operational  picture  must  be  understood  by  a  human  mind. 
The  introduction  of  even  more  complex  COP  models  such  as  the  single  integrated  air  picture 
(SIAP)  imposes  an  even  greater  risk  of  a  ‘complete  understanding  by  the  human’  of  possibly 
hundreds  of  tracks.  Why  do  we  say  this?  Unfortunately  there  are  still  blue  on  blue  casualties, 
mistakes  such  as  the  bombing  of  the  Chinese  embassy  in  Belgrade,  the  shooting  of  the  Iranian 
Airbus,  and  civilian  casualties  due  to  either  poor  combat  identification  or  unreliable  data  sources 
which  create  unverifiable  models.  Risk  reduction,  in  the  sense  of  reducing  if  not  eliminating 
surprise,  again  can  be  split  into  two  distinct  approaches.  The  first  is  to  try  systematically  to  think 
of  all  the  things  that  could  possibly  go  wrong,  and  the  second  is  to  put  in  place  strategies  to 
minimize  the  likelihood  of  error3.  Augmented  cognition  architectures  contain  built  in  mitigation 
mechanisms  which  alter  the  mental  model  inside  the  human  mind.  This  feature,  described  in  the 
architecture  section  below,  offers  tremendous  promise  in  reducing  the  mental  errors  due  to 
misunderstanding  the  mental  model  which  may  have  formed  inside  the  mind  but  may  not 
actually  align  with  reality. 

The  hypothesis  of  this  paper  is  as  follows:  Uncertainty  and  inconsistency  during  complex 
endeavors  can  be  reduced  through  the  application  of  augmented  cognition.  An  analysis  of  the 
genealogy  of  modem  decision  aides  leads  one  to  conclude  that  we  should  only  be  discussing  the 
capability  spectrum  of  intelligent  software  agents.  We  believe  that  this  represents  a  limited  view 
of  the  field  of  automated  decision  aides  and  assisted  cognition.  Instead  of  asking  how  smart  the 
software  agents  can  become,  we  would  like  to  propose  the  following  question:  Can  we  make  the 
human  being  smarter?  Is  it  possible  to  improve  cognitive  functions  inside  the  mind  resulting  in 
better  selections  of  decision  alternatives  and  interpretations  of  events  particularly  in  the  face  of 
growing  amounts  of  data  and  ever  decreasing  time  to  understand  that  data?  Is  it  possible  to 
improve  education,  training  and  warfighter  development?  The  authors  believe  that  the  processes 
and  tools  being  developed  in  the  emerging  field  of  augmented  cognition  can  be  exploited  to 
provide  a  novel  fusion  of  more  capable  human  beings  and  exotic  software  agents.  This  fusion 
should  result  in  breakthrough  levels  of  superior  decision  making  in  environments  of  extreme 
uncertainty.  A  fundamental  goal  of  this  research  was  best  stated  by  Teilhard  de  Chardin:  “The 
fundamental  evil  that  besets  us. .  .is  our  incapacity  to  see  the  whole”4.  We  believe  that  augmented 
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cognition  can  artificially  enhance  human  evolution  to  a  point  where  very  complex  mental  models 
are  understood  in  all  aspects,  thus  enabling  superior  decisions  since  we  will  be  able  to  ‘see  the 
whole’. 

Problem  Statement  and  Discussion 

There  has  been  much  written  in  the  Network  Centric  Warfare  (NCW)  literature 
concerning  situational  awareness  as  it  relates  to  the  ability  to  make  better  decisions.  We  would 
like  to  borrow  the  following  definition  of  situational  awareness: 

Wickens5  defines  situational  awareness  as  “the  continuous  extraction  of  infonnation  about  a 
dynamic  system  or  environment,  the  integration  of  this  information  with  previously  acquired 
knowledge  to  form  a  coherent  mental  picture,  and  the  use  of  that  picture  in  directing  further 
perception  of,  anticipation  of,  and  attention  to  current  events” 

Please  note  that  this  situational  awareness  task  is  currently  being  performed  by  humans  and 
computers  in  tandem,  usually  consisting  of  humans  absorbing  graphical  information  in  a 
common  operational  or  common  tactical  picture.  We  believe  that  the  set  of  mental  activities 
involved  in  arriving  at  the  proper  level  of  situational  awareness  required  to  perfonn  military 
planning  tasks  can  be  greatly  enhanced  by  augmented  cognition  techniques.  Thus,  we  are 
including  situational  awareness  as  an  integral  part  of  the  problem  statement  which  is  as  follows: 
Can  we  directly  embellish  the  human  cognitive  apparatus  involved  in  acquiring  situational 
awareness,  perfonning  complex  planning  tasks,  understanding  common  operational  models,  and 
decision  making  tasks? 

There  is  a  restriction6  on  the  amount  of  mental  tasks  a  military  planner  or  decision  maker 
can  manage  when  solving  planning  problems.  There  is  also,  at  the  moment  a  restriction  on  just 
how  efficient  so  called  ‘intelligent  planning  systems’  can  become.  Creating  an  intelligent  system 
to  aid  in  this  problem-solving  task  is  difficult  because  of  the  constant  flux  of  data  and 
knowledge;  thus,  the  military  planner  or  decision  maker  poses  a  challenge  to  intelligent  systems 
design  and  a  new  model  needs  to  be  created  to  handle  such  problem-solving  issues.  To  create  a 
problem-solving  system  for  such  an  environment,  one  needs  to  consider  the  planners  and  their 
environment,  in  order  to  determine  how  humans  function  in  such  conditions. 

Why  we  believe  that  standalone  AI  (Artificial  Intelligence)  systems  or  un-integrated  human 
beings  alone  are  inadequate 

The  purpose  of  developing  augmented  cognition/enhanced  human  perfonnance  technology  is  to: 

1 .  Create  the  conditions  necessary  for  an  evolutionary  leap  in  the  emergence  of  humans 
with  superior  situational  awareness  capabilities  and  superior  decision  making  skills. 

2.  Improve  asymmetric  thinking,  a  capability  not  currently  possessed  by  artificially 
intelligent  systems. 

3.  Develop  intuitive  decision  making,  also  a  capability  not  currently  possessed  by 
artificially  intelligent  systems. 

4.  Recognize  non-obvious  relationships,  a  capability  possessed  by  current  AI  systems 
but  not  well  demonstrated  by  human  beings 
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5.  Develop  dominant  speed  of  pattern  recognition,  this  capability  is  performed 
adequately  by  both  AI  systems  and  human  beings,  but  in  order  to  achieve 
‘dominance’  much  training  is  required. 

6.  Enhance  intellectual  maneuvering;  a  capability  we  believe  could  be  enabled  by  a 
marriage  of  intelligent  agents  and  non-intrusive  human-computer  direct  integration. 

Until  artificial  intelligence  reaches  a  greater  level  of  maturity,  and  can  exhibit  intuition,  concept 
integration,  or  can  perform  situational  awareness  per  the  Wicken’s  model,  we  believe  that  human 
beings  offer  a  fertile  basis  for  cognitive  research.  But  on  the  other  hand,  human  beings  cannot 
currently  process  large  amounts  of  data;  we  humans  have  well  documented  limitations  in 
attention,  memory,  learning,  comprehension,  sensory  bandwidth,  visualization  abilities, 
qualitative  judgments,  serial  processing  and  decision  making.  However  a  successful  integration 
of  current  intelligent  agent  (AI  based)  software  and  the  human  cognitive  apparatus  can  possibly 
create  human  beings  which  exhibit  superior  cognitive  abilities  with  respect  to  AI  or  existing 
human  cognition.  We  believe  that  the  marriage  of  AI  and  human  beings  can  alleviate  several  of 
the  following  limitations7: 

1 .  Solving  complex  problems  involves  both  implicit  and  explicit  knowledge.  Not  all 
information  is  known  to  the  expert  at  the  time  of  solving  the  problem.  The  expert  needs 
to  search  for  additional  information  in  order  to  solve  the  higher  order  problems.  This  is 
not  an  easy  task  based  on  the  massive  amount  of  data  that  decision  maker  needs  to  sift 
through.  Thus  the  possibility  of  extreme  uncertainty. 

2.  The  amount  of  data,  information,  or  knowledge  required  to  be  analyzed  will  explode 
exponentially  in  the  future  as  pervasive  sensor  systems  are  deployed. 

3.  Solving  complex  problems  involves  solving  sub-problems,  each  of  these  being  complex 
in  nature  and  solution.  Many  AI  systems  cannot  perform  this  task. 

4.  Being  able  to  fuse  information  at  a  sub-problem  level  does  not  necessarily  solve  the 
higher  problem  -  knowledge  compression  must  occur  as  we  progress  up  the  hierarchy  of 
solution  space.  It  is  unclear  that  most  AI  based  systems  can  accomplish  this  function 
without  hardwiring  an  enormous  set  of  rules  or  providing  human  users  with  multiple 
confusing  GUIs  to  inspect  at  a  cost  of  declining  cognitive  perfonnance. 

This  leads  us  to  conclude  that  a  smart  marriage  of  human  cognitive  capabilities  plus  the  data 
processing  and  the  logical  consistency  of  intelligent  agents,  as  part  of  a  well  defined  synergistic 
augmented  cognition  architecture,  will  produce  higher  quality  of  decisions,  the  ability  to  process 
enonnous  amounts  of  data,  and  a  substantial  and  continuously  evolving  improvement  in 
consistency. 

Augmented  Cognition  Description 

What  follows  below  is  an  extensive  set  of  quotations  from  the  experts  in  the  field  of 
augmented  cognition.  These  quotations  describe  augmented  cognition  systems  which  we  believe 
will  facilitate  the  mental  tasks  of  the  planners. 

Limitations  in  human  cognition  are  due  to  intrinsic  restrictions  in  the  number  of  mental  tasks  that 
a  person  can  execute  at  one  time,  and  this  capacity  itself  may  fluctuate  from  moment  to  moment 
depending  on  a  host  of  factors  including  mental  fatigue,  novelty,  boredom  and  stress.  As 
computational  interfaces  have  become  more  prevalent  in  society  and  increasingly  complex  with 
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regard  to  the  volume  and  type  of  information  presented,  researchers  have  investigated  novel 
ways  to  detect  these  bottlenecks  and  have  devised  and  continue  to  detennine  strategies  to  aid 
users  and  improve  their  performance  by  effectively  accommodating  capabilities  and  limitations 

o 

in  human  information  processing  and  decision  making  . 

A  main  goal9  of  the  field  of  Augmented  Cognition  (AugCog)  is  “to  research  and  develop 
technologies  capable  of  extending,  by  an  order  of  magnitude  or  more,  the  information 
management  capacity  of  individuals  working  with  21st  Century  computing  technologies. 

AugCog  science  and  technology  (S&T)  research  and  development  (R&D)  is  therefore  focused  on 
accelerating  the  production  of  novel  concepts  in  human-system  integration  and  includes  the 
study  of  methods  for  addressing  cognitive  bottlenecks  (e.g.,  limitations  in  attention,  memory, 
learning,  comprehension,  visualization  abilities,  and  decision  making)  via  technologies  that 
assess  the  user's  cognitive  status  in  real  time.  A  computational  interaction  employing  such  novel 
system  concepts  monitors  the  state  of  the  user,  through  behavioral,  psychophysiological  and/or 
neurophysiological  data  acquired  from  the  user  in  real  time,  and  then  adapts  or  augments  the 
computational  interface  to  significantly  improve  their  performance  on  the  task  at  hand.” 

Components  of  an  Augmented  Cognition  System10 

“At  the  most  general  level,  the  field  of  Augmented  Cognition  has  the  explicit  goal  of  utilizing 
methods  and  designs  that  harness  computation  and  explicit  knowledge  about  human  limitations 
to  open  bottlenecks  and  address  the  biases  and  deficits  in  human  cognition.  It  proposes  to  do  this 
through  continual  background  sensing,  learning,  and  inferences  to  understand  trends,  patterns, 
and  situations  relevant  to  a  user’s  context  and  goals.  At  its  most  basic  level,  an  augmented 
cognition  system  should  contain  at  least  four  components  -  sensors  for  determining  user  state,  an 
inference  engine  or  classifier  to  evaluate  incoming  sensor  infonnation,  an  adaptive  user  interface, 
and  an  underlying  computational  architecture  to  integrate  these  components.  In  reality  a  fully 
functioning  system  would  have  many  more  components,  but  these  are  the  most  critical  for 
inclusion  as  an  augmented  cognition  system.  Independently,  each  of  these  components  is  fairly 
straightforward.  Much  of  the  ongoing  augmented  cognition  research  focuses  on  integrating  these 
components  to  “close  the  loop,”  and  create  computational  systems  that  adapt  to  their  users.” 

Example  of  a  Possible  Augmented  Cognition  Architecture 

How  would  we  marry  the  capabilities  of  intelligent  agents  and  human  cognition?  In  order  to 
solve  the  awareness  improvement  issues,  data  overload  issues,  and  concept  integration  issues,  we 
need  a  design  that  can  be  implemented  without  user  discomfort.  The  diagram  below  depicts  the 
closed  loop  approach  to  augmented  cognition.  Given  that  the  human  is  interfaced  by  either  an 
eeg  like  hat  or  directly  through  invasive  procedures,  the  agent  sequence  is  as  follows:  First  the 
task  agent  maintains  a  prioritized  list  of  tasks  that  must  be  executed  by  the  human  and  the  other 
agents,  next  the  cognitive  state  sensor  agents  monitor  key  areas  of  memory,  perception,  and 
awareness  and  determine  if  the  highest  priority  task  is  being  executed,  if  not  the  state  effectors 
agent  manipulates  the  humans  awareness  queue  to  force  the  attention  center  to  that  task,  next  the 
cognitive  state  sensor  agent  determines  if  the  human  is  overloaded  with  data,  or  too  many  high 
priority  tasks,  and  requests  a  mitigation  plan  from  the  mitigation  planning  agent.  The  mitigation 
plan  is  sent  to  the  cognitive  mapping  agent  to  detennine  the  precise  plan  for  adjusting  attention 
and  data  flow  to  the  human,  and  finally  these  “adjustments”  are  implemented  by  the  effectors 
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agent.  If  data  is  missing,  or  the  task  is  poorly  described,  the  mitigation  agent  will  request  data 
from  external  sources  to  pass  to  the  human  in  a  managed  manner. 

Augmented  Cognition  Flowchart 


Figure  1  -  Augmented  Cognition  Architecture11 

Thus,  the  challenge  with  these  systems  is  not  the  sensing  component  (although  researchers  are 
using  increasingly  complex  sensors).  The  primary  challenge  with  these  systems  is  accurately 
predicting/assessing,  from  the  incoming  sensor  information,  the  correct  state  of  the  user  and 
having  the  computer  select  an  appropriate  strategy  to  assist  the  user  at  that  time.  As  discussed  in 
the  first  section,  humans  have  well  documented  limitations  in  attention,  memory,  learning, 
comprehension,  sensory  bandwidth,  visualization  abilities,  qualitative  judgments,  serial 
processing  and  decision  making.  For  an  augmented  cognition  system  to  be  successful  it  must 
identify  at  least  one  of  these  bottlenecks  in  real  time  and  alleviate  it  through  a  performance 
enhancing  mitigation  strategy.  These  mitigation  strategies  are  conveyed  to  the  user  through  the 
adaptive  interface  and  might  involve:  modality  switching  (between  visual,  auditory,  &  haptic), 
intelligent  interruption,  task  negotiation  and  scheduling,  and  assisted  context  retrieval  via  book 
marking.  When  a  user  state  is  correctly  sensed  and  an  appropriate  strategy  chosen  to  alleviate  the 
bottleneck,  the  interface  adapted  to  carry  out  the  strategy  and  the  resulting  sensor  information 
indicates  that  the  aiding  has  worked  -  only  then  has  a  system  “closed  the  loop”  and  successfully 
augmented  the  user’s  cognition.  Also,  when  designing  such  a  system,  individual  medical  and 
learning  parameters  must  be  assessed.  Each  system  will  need  to  be  tailored  to  the  target 
individual  for  comfort  and  facilitation  of  use  in  combat  and  non-combat  environments. 


7 


Applications  of  Augmented  Cognition12 

The  applications  of  Augmented  Cognition  research  are  numerous,  and  although  initial 
investments  in  systems  that  explicitly  monitor  cognitive  state  have  been  sponsored  by  military 
and  defense  agencies,  there  is  an  interest  from  the  commercial  sector  to  develop  augmented 
cognition  systems  for  non-military  applications.  As  mentioned  earlier,  closely  related  work  on 
methods  and  architectures  for  detecting  and  reasoning  about  a  user’s  workload  based  on  such 
information  as  activity  with  computing  systems  and  gaze  have  been  studied  for  non-military 
applications  such  as  commercial  notification  systems  and  communication.  There  has  also  been 
interest  from  civilian  agencies  such  as  NASA  on  the  use  of  methods  for  limiting  workload  and 
managing  information  overload.  Hardware  and  software  manufacturers  are  always  eager  to 
include  technologies  that  make  their  systems  easier  to  use,  and  augmented  cognition  systems 
would  likely  result  in  an  increase  in  worker  productivity  with  a  savings  of  both  time  and  money 
to  companies  that  purchased  these  systems.  In  more  specific  cases,  stressful  jobs  that  involve 
constant  infonnation  overload  from  computational  sources,  like  air  traffic  control  would  also 
benefit  from  technology.  Finally,  the  fields  of  education  and  training  are  the  next  likely  targets 
for  this  technology  once  it  reaches  commercial  viability.  Education  and  training  are  moving 
towards  an  increasingly  computational  medium.  With  distance  learning  in  high  demand, 
educational  systems  will  need  to  adapt  to  this  new  non-human  teaching  interaction  while 
ensuring  quality  of  education.  Augmented  Cognition  technologies  could  be  applied  to 
educational  settings  and  guarantee  students  a  teaching  strategy  that  is  adapted  to  their  style  of 
learning.  Above  all  other  domains,  this  application  of  Augmented  Cognition  could  have  the 
biggest  impact  on  society  at  large.  The  positive  impacts  of  a  non-intrusive  augmented  cognition 
system  would  be  to  raise  the  educational  level  of  society  as  a  whole,  creating  human  populations 
with  higher  average  IQ  scores,  increased  situational  awareness  during  combat  scenarios  reducing 
casualties,  identify  mental  disorders  more  rapidly  so  that  early  diagnosis  and  treatment  can  be 
offered  to  the  afflicted  individual,  and  provide  a  novel  suite  of  medical  tools  to  address  problems 
such  as  attention  deficit  disorder.  The  negative  impacts  of  course  would  be  the  development  of 
intrusive  brain  interfaces  with  connections  which  could  possible  override  individual  judgments 
or  free  will.  A  detailed  discussion  of  these  however  is  outside  the  scope  of  this  short  paper. 

Military  examples  of  augmented  cognition13 

To  alleviate  information  and  cognitive  bottlenecks,  the  Marine  Corps  is  demonstrating  an  interest 
in  augmented  cognition.  Virtual  reality-based  training  systems  now  under  development  use 
computer-generated  visuals  that  enable  Marines  to  practice  their  skills  in  a  variety  of  realistic 
situations.  Augmented  cognition  techniques  allow  researchers  to  observe  trainees  and  fit  the 
level  of  training  to  the  individual.  By  directly  observing  a  trainee's  brain  activity,  test  observers 
can  take  the  training  to  a  more  advanced  level  or  scale  it  back  if  the  trainee  shows  signs  of 
overload. 

The  purpose  of  developing  instrumentation  and  situational  awareness  technology  is  to 
demonstrate  affordable  capability  to  enhance  close  quarters  battle  (CQB)  and  military  operations 
on  urbanized  terrain  (MOUT)  training.  This  capability  will  focus  on  a  portable  solution  that  can 
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provide  dismounted  situational  awareness  and  tracking  instrumentation  for  small  unit  training. 
The  benefits  include: 


1.  Enhanced  situational  awareness  capability  for  small  units.  Thus  addressing  data  and 
cognitive  overload  during  periods  of  high  stress. 

2.  Portable,  non-virtual  MOUT  and  CQB  training  capability  for  supporting  improved 
mission  planning  and  rehearsal.  Reducing  the  time  for  data  and  concept  integration. 

Navy  designers  made  use  of  augmented  cognition  methods  as  they  developed  the  control 
interface  for  the  Tomahawk  cruise  missile.  "We  had  our  guys  wear  'brain  caps'  as  they  tested  the 
control  interface,"  said  Schmorrow14.  ”  Instead  of  relying  on  the  users'  subjective  evaluation  of 
how  easy  it  was  to  leam  the  control  system,  researchers  could  observe  their  brain  activity  directly 
and  identify  the  situations  in  which  the  users  became  overwhelmed  with  information”15. 

Certification  and  validation  of  augmented  cognition  as  a  serious  endeavor16 

In  June  2007  the  Human  Factors  and  Ergonomics  Society  (HFES)  Executive  Council  officially 
approved  the  new  Augmented  Cognition  Technical  Group  (AC-TG). 

The  AC-TG  is  concerned  with  fostering  the  development  and  application  of: 

•  real-time  physiological  and  neurophysiological  sensing  technologies  that  can  ascertain  a 
human’s  cognitive  state  while  interacting  with  computing-based  systems; 

•  data  classification  and  integration  architectures  that  enable  closed-loop  system 
applications; 

•  mitigation  (adaptive)  strategies  that  enable  efficient  and  effective  system  adaptation 
based  on  a  user’s  dynamically  changing  cognitive  state; 

•  individually-  tailored  training  systems;  and 

•  roadmaps  for  future  directions  concerning  Augmented  Cognition  (AugCog)  science  and 
technology  (S&T)  and  guidelines  of  use  for  the  technology  and  user  information  that  may 
be  garnered  from  it. 

This  HFES  certification  should  enhance  the  transfer  of  this  technology  to  the  general  public  and 
facilitate  a  broader  acceptance  of  augmented  cognition  by  the  scientific  community  as  a  whole. 


Conclusion 

It  may  be  a  wiser  approach  to  process  the  deluge  of  new  information  with  better  human 
cognition  than  waiting  for  AI  to  mimic  the  human  brain  completely.  It  took  approximately  30 
years  from  serious  chess  program  inception  to  the  defeat  of  Kramnik  last  year  by  Deep  Fritz17. 
Deep  Fritz  still  only  performs  one  function:  playing  chess.  Why  not  use  the  inherent 
adaptability  of  humans  as  an  integral  part  of  our  next  generation  smart  machines.  Given  the 
ever  expanding  flood  of  new  knowledge,  the  military  is  on  the  verge  of  requiring  smarter, 
more  aware,  and  more  focused  human  beings  in  order  to  be  able  to  understand  and  integrate 
all  this  information.  The  push  to  have  intelligent  planning  agents  through  the  use  of  AI  has 
been  an  ongoing  effort  for  many  years.  But  this  paper  is  advocating  that  the  human  should  not 
become  disengaged  from  the  loop  but  rather  be  made  smarter.  The  field  of  augmented 
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cognition  is  opening  many  new  vistas  into  the  mind  and  also  creating  creative  new  concepts 
for  modern  command  and  control.  The  implications  for  the  military  are  enormous  in  that  each 
human  being  can  be  treated  as  a  Network  Centric  node.  Nodal  capabilities  can  then  be 
catalogued  and  accessed  on  demand  possibly  reducing  the  number  of  physical  planning  assets 
required,  increasing  situational  awareness  by  being  able  to  access  nodal  awareness  at  the 
human  mental  picture  level,  data  fusion  at  the  nodal  level,  reduced  blue  on  blue  casualties, 
reduced  civilian  casualties,  improved  prediction  of  combat  fatigue  and  prediction  of  the 
likelihood  of  a  particular  soldier  to  develop  mental  issues  such  as  post  traumatic  stress 
syndrome.  By  using  the  cognitively  augmented  war  fighters  as  'live  nodes'  on  the  GIG,  this 
would  permit  general  access  to  improved  skills  and  globally  fused  local  situational 
awareness  models. 
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Introduction 


•  The  hypothesis  of  this  paper  is  as  follows:  Uncertainty  and  inconsistency  during 
complex  endeavors  can  be  reduced  through  the  application  of  augmented  cognition. 

•  An  analysis  of  the  genealogy  of  modern  decision  aides  leads  one  to  conclude  that 
we  should  only  be  discussing  the  capability  spectrum  of  intelligent  software  agents. 

•  We  believe  that  this  represents  a  limited  view  of  the  field  of  automated  decision 
aides  and  assisted  cognition. 

•  Instead  of  asking  how  smart  the  software  agents  can  become,  we  would  like  to 
propose  the  following  question: 

•  Can  we  make  the  human  being  smarter? 

•  Is  it  possible  to  improve  cognitive  functions  inside  the  mind  resulting  in  better 
selections  of  decision  alternatives  and  interpretations  of  events? 

•  Is  it  possible  to  radically  alter  human  training,  development,  and  education  to 
optimize  the  potential  of  every  individual? 

•  The  authors  believe  that  the  processes  and  tools  being  developed  in  the  emerging 
field  of  augmented  cognition  1  can  be  exploited  to  provide  a  novel  fusion  of  more 
capable  human  beings  and  exotic  software  agents. 

•  This  fusion  should  result  in  breakthrough  levels  of  situational  awareness  and 
superior  decision  making  in  environments  of  extreme  uncertainty. 


Connecting  to  the  brain  directly 


Augmented  cognition  speaks  of  directly  connecting  parts  of  the  brain  to  external 
interfaces  where  agents  can  monitor  internal  cognitive  states,  create  'external 
mental  models',  evaluate  those  models  and  states  and  request  a  re-prioritization 
of  certain  cognitive  and  mental  activities  back  to  the  brain  of  the  human  user. 

We  consider  agents  capable  of  modifying  internal  human  cognitive  processes  to 
be  exotic  by  definition. 

Uncertainty  in  decision  making  has  been  analyzed  in  some  detail. 

The  two  types  of  uncertainty  which  we  are  interested  in  reducing  are  aleatoric  and 
epistemic. 

While  only  multiple,  diverse,  and  rich  courses  of  action  can  manage  aleatoric 
uncertainty,  epistemic  uncertainty  arises  from  the  quality  of  a  model;  that  is,  the 
extent  to  which  a  model,  because  of  its  necessary  assumptions  and 
approximations,  does  not  truly  represent  that  which  is  being  modeled2. 

Situational  awareness,  a  key  aspect  of  decision  making,  is  based  upon  many 
models,  for  example,  a  common  operational  picture,  a  common  tactical  picture, 
etc.  all  of  which  are  based  upon  the  assumption  of  timeliness,  validity,  and 
understandability  of  track  data  and  its  sources.  But  above  all,  the  common 
operational  picture  must  be  understood  by  a  human  mind. 


Situational  Awareness  Definition 


Wickens  defines  situational  awareness  as  "the 
continuous  extraction  of  information  about  a 
dynamic  system  or  environment,  the 
integration  of  this  information  with  previously 
acquired  knowledge  to  form  a  coherent 
mental  picture,  and  the  use  of  that  picture  in 
directing  further  perception  of,  anticipation 
of,  and  attention  to  current  events" 


Problem  Statement 


Can  we  directly  embellish  the  human 
cognitive  apparatus  involved  in  acquiring 
situational  awareness,  performing  complex 
planning  tasks,  understanding  common 
operational  models,  and  decision  making 
tasks? 


Why  we  believe  that  standalone  AI  (Artificial  Intelligence) 
systems  or  un-integrated  human  beings  alone  are  inadequate 


Until  artificial  intelligence  reaches  a  greater  level  of  maturity,  and  can  exhibit  intuition,  concept  integration, 
or  can  perform  situational  awareness  per  the  Wicken’s  model,  we  believe  that  human  beings  offer  a  fertile 
basis  for  cognitive  research.  But  on  the  other  hand,  human  beings  cannot  currently  process  large  amounts  of 
data;  we  humans  have  well  documented  limitations  in  attention,  memory,  learning,  comprehension,  sensory 
bandwidth,  visualization  abilities,  qualitative  judgments,  serial  processing  and  decision  making 

We  believe  that  the  marriage  of  AI  and  human  beings  can  alleviate  several  of  the  following  limitations: 

-  Solving  complex  problems  involves  both  implicit  and  explicit  knowledge.  Not  all  information  is 
known  to  the  expert  at  the  time  of  solving  the  problem.  The  expert  needs  to  search  for  additional 
information  in  order  to  solve  the  higher  order  problems.  This  is  not  an  easy  task  based  on  the  massive 
amount  of  data  that  decision  maker  needs  to  sift  through.  Thus  the  possibility  of  extreme  uncertainty. 

-  The  amount  of  data,  information,  or  knowledge  required  to  be  analyzed  will  explode  exponentially  in 
the  future  as  pervasive  sensor  systems  are  deployed. 

-  Solving  complex  problems  involves  solving  sub-problems,  each  of  these  being  complex  in  nature  and 
solution.  Many  AI  systems  cannot  perform  this  task. 

-  Being  able  to  fuse  information  at  a  sub-problem  level  does  not  necessarily  solve  the  higher  problem  - 
knowledge  compression  must  occur  as  we  progress  up  the  hierarchy  of  solution  space.  It  is  unclear 
that  most  AI  based  systems  can  accomplish  this  function  without  hardwiring  an  enormous  set  of  rules 
or  providing  human  users  with  multiple  confusing  GUIs  to  inspect  at  a  cost  of  declining  cognitive 
performance. 

This  leads  us  to  conclude  that  a  smart  marriage  of  human  cognitive  capabilities  plus  the  data  processing 
and  the  logical  consistency  of  intelligent  agents,  as  part  of  a  well  defined  synergistic  augmented  cognition 
architecture,  will  produce  higher  quality  of  decisions,  the  ability  to  process  enormous  amounts  of  data, 
and  a  substantial  and  continuously  evolving  improvement  in  consistency. 


Goals  of  Augmented  Cognition 


Create  the  conditions  necessary  for  an  evolutionary  leap  in  the 
emergence  of  humans  with  superior  situational  awareness 
capabilities  and  superior  decision  making  skills. 

Improve  asymmetric  thinking,  a  capability  not  currently  possessed 
by  artificially  intelligent  systems. 

Develop  intuitive  decision  making,  also  a  capability  not  currently 
possessed  by  artificially  intelligent  systems. 

Recognize  non-obvious  relationships,  a  capability  possessed  by 
current  AI  systems  but  not  well  demonstrated  by  human  beings 

Develop  dominant  speed  of  pattern  recognition,  this  capability  is 
performed  adequately  by  both  AI  systems  and  human  beings,  but  in 
order  to  achieve  ‘dominance’  much  training  is  required. 

Enhance  intellectual  maneuvering;  a  capability  we  believe  could  be 
enabled  by  a  marriage  of  intelligent  agents  and  non-intrusive 
human-computer  direct  integration 


Components  of  an  Augmented 
Cognition  System 

At  its  most  basic  level,  an  augmented  cognition  system  should 
contain  at  least  four  components: 

sensors  for  determining  user  state 

An  inference  engine  or  classifier  to  evaluate  incoming  sensor 
information 

An  adaptive  user  interface, 

An  underlying  computational  architecture  to  integrate  these 
components. 

In  reality  a  fully  functioning  system  would  have  many  more 
components,  but  these  are  the  most  critical  for  inclusion  as  an 
augmented  cognition  system.  Independently,  each  of  these 
components  is  fairly  straightforward.  Much  of  the  ongoing 
augmented  cognition  research  focuses  on  integrating  these 
components  to  "close  the  loop,"  and  create  computational  systems 
that  adapt  to  their  users." 


Augmented  Cognition  Architecture 

Augmented  Cognition  Flowchart 


Applications  of  Augmented  Cognition 


There  has  also  been  interest  from  civilian  agencies  such  as  NASA  on  the  use  of  methods  for  limiting 
workload  and  managing  information  overload. 

Hardware  and  software  manufacturers  are  always  eager  to  include  technologies  that  make  their  systems 
easier  to  use,  and  augmented  cognition  systems  would  likely  result  in  an  increase  in  worker  productivity 
with  a  savings  of  both  time  and  money  to  companies  that  purchased  these  systems. 

In  more  specific  cases,  stressful  jobs  that  involve  constant  information  overload  from  computational 
sources,  like  air  traffic  control  would  also  benefit  from  technology. 

The  fields  of  education  and  training  are  the  next  likely  targets  for  this  technology  once  it  reaches 
commercial  viability.  Education  and  training  are  moving  towards  an  increasingly  computational  medium. 
With  distance  learning  in  high  demand,  educational  systems  will  need  to  adapt  to  this  new  non-human 
teaching  interaction  while  ensuring  quality  of  education.  Augmented  Cognition  technologies  could  be 
applied  to  educational  settings  and  guarantee  students  a  teaching  strategy  that  is  adapted  to  their  style  of 
learning. 

Above  all  other  domains,  this  application  of  Augmented  Cognition  could  have  the  biggest  impact  on 
society  at  large.  The  positive  impacts  of  a  non-intrusive  augmented  cognition  system  would  be  to  raise  the 
educational  level  of  society  as  a  whole,  creating  human  populations  with  higher  average  IQ  scores, 
increased  situational  awareness  during  combat  scenarios  reducing  casualties,  identify  mental  disorders 
more  rapidly  so  that  early  diagnosis  and  treatment  can  be  offered  to  the  afflicted  individual,  and  provide  a 
novel  suite  of  medical  tools  to  address  problems  such  as  attention  deficit  disorder. 


Military  Examples  of  Augmented  Cognition 


To  alleviate  information  and  cognitive  bottlenecks,  the  Marine  Corps  is  demonstrating  an 
interest  in  augmented  cognition.  Virtual  reality-based  training  systems  now  under 
development  use  computer-generated  visuals  that  enable  Marines  to  practice  their  skills  in  a 
variety  of  realistic  situations.  Augmented  cognition  techniques  allow  researchers  to  observe 
trainees  and  fit  the  level  of  training  to  the  individual.  By  directly  observing  a  trainee's  brain 
activity,  test  observers  can  take  the  training  to  a  more  advanced  level  or  scale  it  back  if  the 
trainee  shows  signs  of  overload. 

The  purpose  of  developing  instrumentation  and  situational  awareness  technology  is  to 
demonstrate  affordable  capability  to  enhance  close  quarters  battle  (CQB)  and  military 
operations  on  urbanized  terrain  (MOUT)  training.  This  capability  will  focus  on  a  portable 
solution  that  can  provide  dismounted  situational  awareness  and  tracking  instrumentation  for 
small  unit  training.  The  benefits  include: 

Enhanced  situational  awareness  capability  for  small  units.  Thus  addressing  data  and  cognitive 
overload  during  periods  of  high  stress. 

Portable,  non-virtual  MOUT  and  CQB  training  capability  for  supporting  improved  mission 
planning  and  rehearsal.  Reducing  the  time  for  data  and  concept  integration. 

Navy  designers  made  use  of  augmented  cognition  methods  as  they  developed  the  control 
interface  for  the  Tomahawk  cruise  missile.  "We  had  our  guys  wear  'brain  caps'  as  they  tested 
the  control  interface,"  said  Schmorrow14.  ”  Instead  of  relying  on  the  users'  subjective 
evaluation  of  how  easy  it  was  to  learn  the  control  system,  researchers  could  observe  their 
brain  activity  directly  and  identify  the  situations  in  which  the  users  became  overwhelmed  with 
information”15. 


Certification  and  Validation  of  Augmented  Cognition  as  a 

Serious  Endeavor 


In  June  2007  the  Human  Factors  and  Ergonomics  Society  (HFES)  Executive 
Council  officially  approved  the  new  Augmented  Cognition  Technical  Group 
(AC-TG). 

The  AC-TG  is  concerned  with  fostering  the  development  and  application 
of: 

-  real-time  physiological  and  neurophysiological  sensing  technologies  that  can 
ascertain  a  human's  cognitive  state  while  interacting  with  computing-based 
systems; 

-  data  classification  and  integration  architectures  that  enable  closed-loop 
system  applications; 

-  mitigation  (adaptive)  strategies  that  enable  efficient  and  effective  system 
adaptation  based  on  a  user's  dynamically  changing  cognitive  state; 

-  individually-  tailored  training  systems;  and 

-  roadmaps  for  future  directions  concerning  Augmented  Cognition  (AugCog) 
science  and  technology  (S&T)  and  guidelines  of  use  for  the  technology  and 
user  information  that  may  be  garnered  from  it. 


Conclusion 


It  may  be  a  wiser  approach  to  process  the  deluge  of 
new  information  with  better  human  cognition  than 
waiting  for  Al  to  mimic  the  human  brain  completely.  It 
took  approximately  30  years  from  serious  chess 
Drogram  inception  to  the  defeat  of  Kramnik  last  year 
Dy  Deep  Fritz17.  Deep  Fritz  still  only  performs  one 
function:  playing  chess.  Why  not  use  the  inherent 
adaptability  of  humans  as  an  integral  part  of  our  next 
generation  smart  machines.  Given  the  ever  expanding 
flood  of  new  knowledge,  the  military  is  on  the  verge  of 
requiring  smarter,  more  aware,  and  more  focused 
human  beings  in  order  to  be  able  to  understand  and 
integrate  all  this  information. 


